<?xml version="1.0" encoding="iso-8859-1"?>
<chapter>

<title>Using Regular Expressions to Specify Simple Datatypes</title>


<para>Among  the
different 
facets
available to
restrict the lexical space of simple datatypes,
the most flexible (and also the one that we will often use as a last
resort when all the other facets are unable to express the
restriction on a user-defined datatype) is based on regular
expressions.
</para>

<note><para>There is a terminology clash between Relax NG's patterns and the "pattern" facet of W3C XML Schema. To limit the risk of confusion we will refer to the facet as the "pattern facet" or "regular expression".</para></note>


<sect1>
<title>The Swiss Army Knife</title>

<para>The pattern facet (and regular
expressions in general) is like a Swiss
army knife when constraining simple datatypes. It is highly
flexible, can compensate for many of the limitations of the other
facets, and are often used to define user datatypes on various
formats such as ISBN numbers, telephone numbers, or custom date
formats. However, like a Swiss army knife, it has its own
limitation.
</para>

<para>Cutting a tree with a Swiss army knife is long, tiring, and
dangerous. Writing regular expressions may also become long, tiring,
and dangerous when the number of combinations grows. One should try
to keep them as simple as possible.
</para>

<para>A Swiss army knife cannot change lead into gold, and no facet can
change the primary type of a simple datatype. A string datatype
restricted to match a custom date format will still retain the
properties of a string and will never acquire the facets of a
datetime datatype. This means that there is no effective way to
express localized date formats.
</para>

</sect1>

<sect1>
<title>The Simplest Possible Pattern facets</title>

<para>In their simplest form,
pattern facets may be used as enumerations
applied to the lexical space rather than on the value space.
</para>

<para>If, for instance, we have a byte value that can only take the values
&quot;1,&quot;
&quot;5,&quot; or
&quot;15,&quot; the classical way to define
such a datatype is to use the Relax NG's <literal>choice</literal> pattern:
</para>

<programlisting><![CDATA[
      <choice>
        <value type="byte">1</value>
        <value type="byte">5</value>
        <value type="byte">15</value>
      </choice>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
  element foo {
  xsd:byte "1"
  | xsd:byte "5"
  | xsd:byte "15"
 }
]]></programlisting>


<para>This is the &quot;normal&quot; way of
defining this datatype if it matches the lexical space and the value
space of an <literal>xsd:byte</literal>. It gives
the flexibility to accept the instance documents with values such as
&quot;1,&quot;
&quot;5,&quot; and
&quot;15,&quot; but also
&quot;01&quot; or
&quot;0000005.&quot;
</para>
<para>As far as validation only is concerned, if we wanted to remove the variations with leading zeros, we could just use another datatype such as <literal>token</literal> instead of <literal>xsd:byte</literal> in our <literal>choice</literal> pattern:</para>

<programlisting><![CDATA[
      <choice>
        <value type="token">1</value>
        <value type="token">5</value>
        <value type="token">15</value>
      </choice>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
  xsd:token "1"
  | xsd:token "5"
  | xsd:token "15"
]]></programlisting>

<para>However, we might have good reasons to use a  <literal>xsd:byte</literal>, for instance if we are interested in type annotation and want that a Relax NG processor supporting type annotation reports the datatype as <literal>xsd:byte</literal> and not <literal>xsd:token</literal>.</para>

<para>One of the
particularities of the pattern facet is it must be the only facet
constraining the lexical space. If we have an application that is
disturbed by leading zeros, we can use pattern facets instead of
enumerations to define our datatype:
</para>

<programlisting><![CDATA[
      <data type="byte">
        <param name="pattern">1|5|15</param>
      </data>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
 xsd:byte {pattern = "1|5|15"}
]]></programlisting>

<para>Here, we are still using the <literal>xsd:byte</literal> datatype with its semantic, but
its lexical space is now constrained to accept only
&quot;1,&quot;
&quot;5,&quot; and
&quot;15,&quot; leaving out any variation
that has the same value but a different lexical representation.
</para>



<tip>
<para>This is an important difference from Perl regular expressions, on
which W3C XML Schema pattern facets are built.
A Perl expression such as <literal>/15/</literal> matches any string
containing &quot;15,&quot; while the W3C XML
Schema pattern facet matches only the string equal to
&quot;15.&quot; The Perl expression
equivalent to this pattern facet is thus <literal>/^15$/</literal>.
</para>
</tip>

<para>This example has been carefully chosen to avoid using any of the meta
characters used within pattern facets, which are:
&quot;.&quot;,
&quot;\&quot;,
&quot;?&quot;,
&quot;*&quot;,
&quot;+&quot;,
&quot;{&quot;,
&quot;}&quot;,
&quot;(&quot;,
&quot;)&quot;,
&quot;[&quot;, and
&quot;]&quot;. We will see the meaning of
these characters later in this chapter; for the moment, we just need
to know that each of these   characters needs to be
&quot;escaped&quot; by a leading
&quot;\&quot; to be used as a literal. For
instance, to define a similar datatype for a decimal when lexical
space is limited to &quot;1&quot; and
&quot;1.5,&quot; we write:
</para>

<programlisting><![CDATA[
      <data type="decimal">
        <param name="pattern">1|1\.5</param>
      </data>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
 xsd:decimal {pattern = "1|1\.5"}
]]></programlisting>
<para>A common source of errors is that
&quot;normal&quot; characters should not be
escaped: we will see later that a leading
&quot;\&quot; changes their meaning (for
instance, &quot;\P&quot; matches all the
Unicode punctuation characters and not the character
&quot;P&quot;).
</para>

</sect1>

<sect1>
<title>Quantifying</title>

<para>Despite an
apparent similarity, the pattern facet interprets its value
in a very different way than <literal>value</literal> does. <literal>value</literal> reads the value as a lexical representation,
and converts it to the corresponding value for its base datatype,
while pattern facet reads the value
as a set of conditions to apply on lexical values. When we write:
</para>

<programlisting> pattern=&quot;15&quot;</programlisting>

<para>we specify three conditions (first character equals
&quot;1,&quot; second character equals
&quot;5,&quot; and the string must finish
after this). Each of the matching conditions (such as first character
equals &quot;1&quot; and second character
equals &quot;5&quot;) is called a piece. This
is just the simplest form to specify piece.
</para>

<para>Each piece in a pattern facet is composed of an atom
identifying a
character, or a set of characters, and an optional quantifier.
Characters (except special characters that must be escaped) are the
simplest form of atoms. In our example, we have omitted the
quantifiers. 
Quantifiers may be defined using two
different syntaxes: either a special character (<literal>*</literal>
for 0 or more, <literal>+</literal> for one or more, and
<literal>?</literal> for 0 or 1) or a numeric range within curly
braces (<literal>{n}</literal> for exactly n times,
<literal>{n,m}</literal> for between n and m times, or
<literal>{n,}</literal> for n or more times).
</para>

<para>Using these quantifiers, we can merge our three pattern facets into one:</para>

<programlisting><![CDATA[
      <data type="byte">
        <param name="pattern">1?5?</param>
      </data>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
 xsd:byte {pattern = "1?5?"}
]]></programlisting>

<para>This new pattern facet means there must be zero or one character
(&quot;1&quot;) followed by zero or one
character (&quot;5&quot;). This is not
exactly the same meaning as our three previous pattern facets since the
empty string &quot;&quot; is now accepted by
the pattern facet. However, since the empty string doesn't
belong to the lexical space of our base type (<literal>xsd:byte</literal>), the new datatype has the same
lexical space as the previous one.
</para>

<para>We could also use quantifiers to limit the number of leading
zeros--for instance, the following pattern facet limits the number of
leading zeros to up to 2:
</para>

<programlisting><![CDATA[
      <data type="byte">
        <param name="pattern">0{0,2}1?5?</param>
      </data>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
 xsd:byte {pattern = "0{0,2}1?5?"}
]]></programlisting>

</sect1>

<sect1>
<title>More Atoms</title>

<para>By this point, we have seen the simplest atoms that can be used in a
pattern facet: &quot;1,&quot;
&quot;5,&quot; and
&quot;\.&quot; are atoms that exactly match a
character. The other atoms that can be used in pattern facets are special
characters, a wildcard that matches any character, or predefined and
user-defined character classes.
</para>

<sect2>
<title>Special Characters</title>

<para>Table 6-1 shows the    list of atoms that match a single
character, exactly like the characters we have already seen, but also
correspond to characters that must be escaped or (for the first three
characters on the list) that are just provided for convenience.
</para>

<table>

<title>Special characters</title>
<tgroup cols="2">
<colspec colnum="1" colname="col1"/>
<colspec colnum="2" colname="col2"/>
<tbody>
<row>
<entry>
<para>\n</para>
</entry>
<entry>
<para>New line (can also
be written as &quot;&amp;#x0A;-- since we
are in a XML document).
</para>
</entry>
</row>
<row>
<entry>
<para>\r</para>
</entry>
<entry>
<para>Carriage return (can also be written as
&quot;&amp;#x0D; -- ). 
</para>
</entry>
</row>
<row>
<entry>
<para>\t</para>
</entry>
<entry>
<para>Tabulation (can also be written
as &quot;&amp;#x09; -- )
</para>
</entry>
</row>
<row>
<entry>
<para>\\</para>
</entry>
<entry>
<para>Character
&quot;\&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\|</para>
</entry>
<entry>
<para>Character
&quot;|&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\.</para>
</entry>
<entry>
<para>Character
&quot;.&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\-</para>
</entry>
<entry>
<para>Character
&quot;-&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\^</para>
</entry>
<entry>
<para>Character
&quot;^&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\?</para>
</entry>
<entry>
<para>Character
&quot;?&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\*</para>
</entry>
<entry>
<para>Character
&quot;*&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\+</para>
</entry>
<entry>
<para>Character
&quot;+&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\{</para>
</entry>
<entry>
<para>Character
&quot;{&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\}</para>
</entry>
<entry>
<para>Character
&quot;}&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\(</para>
</entry>
<entry>
<para>Character
&quot;(&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\)</para>
</entry>
<entry>
<para>Character
&quot;)&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\[</para>
</entry>
<entry>
<para>Character
&quot;[&quot;
</para>
</entry>
</row>
<row>
<entry>
<para>\]</para>
</entry>
<entry>
<para>Character
&quot;]&quot;
</para>
</entry>
</row>
</tbody>
</tgroup>
</table>

</sect2>
<sect2>
<title>Wildcard</title>

<para>The     character
&quot;.&quot; has a special meaning:
it's a wildcard atom that matches any XML valid
character except newlines and carriage returns. As with any atom,
&quot;.&quot; may be followed by an optional
quantifier and &quot;.*&quot; is a common
construct to match zero or more occurrences of any character. To
illustrate the usage of &quot;.*&quot; (and
the fact that pattern facet is a Swiss army knife), a
pattern facet may be used to define the integers that are multiples of 10:
</para>

<programlisting><![CDATA[
  <define name="multipleOfTen">
    <data type="integer">
      <param name="pattern">.*0</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>
<programlisting><![CDATA[
  multipleOfTen = xsd:integer {pattern = ".*0"}
]]></programlisting>

</sect2>
<sect2>
<title>Character Classes</title>

<para>W3C   XML Schema has adopted the
&quot;classical&quot; Perl and Unicode
character classes (but not the POSIX-style character classes also
available in Perl).
</para>

<sect3>
<title>Classical Perl character classes</title>

<para>W3C  XML Schema supports the classical Perl
character classes plus a couple of additions to match XML-specific
productions. Each of these classes are designated by a single letter;
the classes designated by the upper- and lowercase versions of the
same letter are complementary:
</para>

<variablelist>
<varlistentry><term><literal>\s</literal></term>
<listitem>
<para>Spaces. Matches the XML whitespaces (space
#x20, tabulation #x09, line feed #x0A, and carriage return #x0D).
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\S</literal></term>
<listitem>
<para>Characters that are not spaces.
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\d</literal></term>
<listitem>
<para>Digits
(&quot;0&quot; to
&quot;9&quot; but also digits in other
alphabets).
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\D</literal></term>
<listitem>
<para>Characters that are not digits.
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\w</literal></term>
<listitem>
<para>Extended
&quot;word&quot; characters (any Unicode
character not defined as
&quot;punctuation&quot;,
&quot;separator,&quot; and
&quot;other&quot;). This conforms to the Perl
definition, assuming UTF8 support has been switched on.
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\W</literal></term>
<listitem>
<para>Nonword characters.
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\i</literal></term>
<listitem>
<para>XML 1.0 initial name characters (i.e., all
the &quot;letters&quot; plus
&quot;-&quot;). This is a W3C XML Schema
extension over Perl regular expressions.
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\I</literal></term>
<listitem>
<para>Characters that may not be used as a XML
initial name character.
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\c</literal></term>
<listitem>
<para>XML 1.0 name characters (initial name
characters, digits, &quot;.&quot;,
&quot;:&quot;,
&quot;-&quot;, and the characters defined by
Unicode as &quot;combining&quot; or
&quot;extender&quot;). This is a W3C XML
Schema extension to Perl regular expressions.
</para>
</listitem>
</varlistentry>

<varlistentry><term><literal>\C</literal></term>
<listitem>
<para>Characters that may not be used in a XML
1.0 name.
</para>
</listitem>
</varlistentry>
</variablelist>

<para>These character classes may be used with an optional quantifier like
any other atom. The last pattern facet that we saw:
</para>

<programlisting>multipleOfTen = xsd:integer {pattern = ".*0"}</programlisting>

<para>constrains the lexical space to be a string of characters ending with
a zero. Knowing that the base type is a
<literal>xsd:integer</literal>, this is good enough for our purposes,
but if the base type had been a <literal>xsd:decimal</literal> (or <literal>xsd:string</literal>), we could be more restrictive and
write:
</para>

<programlisting>multipleOfTen = xsd:integer {pattern = "-?\d*0"}</programlisting>

<para>This checks that the characters before the trailing zero are digits
with an optional leading <literal>-</literal> (we will see later on
in Section 6.5.2.2 how to specify an optional
leading <literal>-</literal> or <literal>+</literal>).
</para>

</sect3>

<sect3>
<title>Unicode character classes</title>

<para>Patterns support character classes matching
both Unicode  

categories
and blocks. Categories and blocks are two complementary
classification systems: categories classify the characters by their
usage independently to their localization (letters, uppercase, digit,
punctuation, etc.), while blocks classify characters by their
localization independently of their usage (Latin, Arabic, Hebrew,
Tibetan, and even Gothic or musical symbols).
</para>

<para>The syntax <literal>\p{Name}</literal> is similar for blocks and
categories; the prefix <literal>Is</literal> is added to the name of
categories to make the distinction. The syntax
<literal>\P{Name}</literal> is also available to select the
characters that do not match a block or category. A list of Unicode
blocks and categories is given in the specification. Table 6-2 shows the Unicode character classes and Table 6-3 shows the Unicode character blocks.
</para>

<table>

<title>Unicode character classes </title>
<tgroup cols="2">
<colspec colnum="1" colname="col1"/>
<colspec colnum="2" colname="col2"/>

<thead>
<row>
<entry>
<para>Unicode Character Class</para>
</entry>
<entry>
<para>Includes</para>
</entry>
</row>
</thead>
<tbody>

<row>
<entry>
<para>C</para>
</entry>
<entry>
<para>Other characters (non-letters, non symbols, non-numbers,
non-separators)
</para>
</entry>
</row>

<row>
<entry>
<para>Cc</para>
</entry>
<entry><para>Control characters</para>
</entry>
</row>

<row>
<entry>
<para>Cf</para>
</entry>
<entry>
<para>Format characters</para>
</entry>
</row>

<row>
<entry>
<para>Cn</para>
</entry>
<entry>
<para>Unassigned code points</para>
</entry>

</row>
<row>
<entry>
<para>Co</para>
</entry>
<entry>
<para>Private use characters</para>
</entry>
</row>

<row>
<entry>
<para>L</para>
</entry>
<entry>
<para>Letters</para>
</entry>
</row>

<row>
<entry>
<para>Ll</para>
</entry>
<entry>
<para>Lowercase letters</para>
</entry>
</row>

<row>
<entry>
<para>Lm</para>
</entry>
<entry>
<para>Modifier letters</para>
</entry>
</row>

<row>
<entry>
<para>Lo</para>
</entry>
<entry>
<para>Other letters</para>
</entry>
</row>

<row>
<entry>
<para>Lt</para>
</entry>
<entry>
<para>Titlecase letters</para>
</entry>
</row>

<row>
<entry>
<para>Lu</para>
</entry>
<entry>
<para>Uppercase letters</para>
</entry>
</row>

<row>
<entry>
<para>M</para>
</entry>
<entry>
<para>All Marks</para>
</entry>
</row>

<row>
<entry>
<para>Mc</para>
</entry>
<entry>
<para>Spacing combining marks</para>
</entry>
</row>

<row>
<entry>
<para>Me</para>
</entry>
<entry>
<para>Enclosing marks</para>
</entry>
</row>

<row>
<entry>
<para>Mn</para>
</entry>
<entry>
<para>Non-spacing marks</para>
</entry>
</row>

<row>
<entry>
<para>N</para>
</entry>
<entry>
<para>Numbers</para>
</entry>
</row>

<row>
<entry>
<para>Nd</para>
</entry>
<entry>
<para>Decimal digits</para>
</entry>
</row>

<row>
<entry>
<para>Nl</para>
</entry>
<entry>
<para>Number letters</para>
</entry>
</row>


<row>
<entry>
<para>No</para>
</entry>
<entry>
<para>Other numbers</para>
</entry>
</row>

<row>
<entry>
<para>P</para>
</entry>
<entry>
<para>Punctuation</para>
</entry>
</row>

<row>
<entry>
<para>Pc</para>
</entry>
<entry>
<para>Connector punctuation</para>
</entry>
</row>

<row>
<entry>
<para>Pd</para>
</entry>
<entry>
<para>Dashes</para>
</entry>
</row>

<row>
<entry>
<para>Pe</para>
</entry>
<entry>
<para>Closing punctuation</para>
</entry>
</row>

<row>
<entry>
<para>Pf</para>
</entry>
<entry>
<para>Final quotes (may behave like Ps or Pe)</para>
</entry>
</row>

<row>
<entry>
<para>Pi</para>
</entry>
<entry>
<para>Initial quotes (may behave like Ps or Pe)</para>
</entry>
</row>

<row>
<entry>
<para>Po</para>
</entry>
<entry>
<para>Other forms of punctuation</para>
</entry>
</row>

<row>
<entry>
<para>Ps</para>
</entry>
<entry>
<para>Opening punctuation</para>
</entry>
</row>



<row>
<entry>
<para>S</para>
</entry>
<entry>
<para>Symbols</para>
</entry>
</row>

<row>
<entry>
<para>Sc</para>
</entry>
<entry>
<para>Currency symbols</para>
</entry>
</row>

<row>
<entry>
<para>Sk</para>
</entry>
<entry>
<para>Modifier symbols</para>
</entry>
</row>

<row>
<entry>
<para>Sm</para>
</entry>
<entry>
<para>Mathematical symbols</para>
</entry>
</row>

<row>
<entry>
<para>So</para>
</entry>
<entry>
<para>Other symbols</para>
</entry>
</row>

<row>
<entry>
<para>Z</para>
</entry>
<entry>
<para>Separators</para>
</entry>
</row>

<row>
<entry>
<para>Zl</para>
</entry>
<entry>
<para>Line breaks</para>
</entry>
</row>

<row>
<entry>
<para>Zp</para>
</entry>
<entry>
<para>Paragraph breaks</para>
</entry>
</row>

<row>
<entry>
<para>Zs</para>
</entry>
<entry>
<para>Spaces</para>
</entry>
</row>
</tbody>
</tgroup>
</table>

<table>

<title>Unicode character blocks</title>
<tgroup cols="3">
<colspec colnum="1" colname="col1"/>
<colspec colnum="2" colname="col2"/>
<colspec colnum="3" colname="col3"/>
<tbody>
<row>
<entry>
<para>AlphabeticPresentationForms</para>
</entry>
<entry>
<para>Arabic</para>
</entry>
<entry>
<para>ArabicPresentationForms-A</para>
</entry>
</row>
<row>
<entry>
<para>ArabicPresentationForms-B</para>
</entry>
<entry>
<para>Armenian</para>
</entry>
<entry>
<para>Arrows</para>
</entry>
</row>
<row>
<entry>
<para>BasicLatin</para>
</entry>
<entry>
<para>Bengali</para>
</entry>
<entry>
<para>BlockElements</para>
</entry>
</row>
<row>
<entry>
<para>Bopomofo</para>
</entry>
<entry>
<para>BopomofoExtended</para>
</entry>
<entry>
<para>BoxDrawing</para>
</entry>
</row>
<row>
<entry>
<para>BraillePatterns</para>
</entry>
<entry>
<para>ByzantineMusicalSymbols</para>
</entry>
<entry>
<para>Cherokee</para>
</entry>
</row>
<row>
<entry>
<para>CJKCompatibility</para>
</entry>
<entry>
<para>CJKCompatibilityForms</para>
</entry>
<entry>
<para>CJKCompatibilityIdeographs</para>
</entry>
</row>
<row>
<entry>
<para>CJKCompatibilityIdeographsSupplement</para>
</entry>
<entry>
<para>CJKRadicalsSupplement</para>
</entry>
<entry>
<para>CJKSymbolsandPunctuation</para>
</entry>
</row>
<row>
<entry>
<para>CJKUnifiedIdeographs</para>
</entry>
<entry>
<para>CJKUnifiedIdeographsExtensionA</para>
</entry>
<entry>
<para>CJKUnifiedIdeographsExtensionB</para>
</entry>
</row>
<row>
<entry>
<para>CombiningDiacriticalMarks</para>
</entry>
<entry>
<para>CombiningHalfMarks</para>
</entry>
<entry>
<para>CombiningMarksforSymbols</para>
</entry>
</row>
<row>
<entry>
<para>ControlPictures</para>
</entry>
<entry>
<para>CurrencySymbols</para>
</entry>
<entry>
<para>Cyrillic</para>
</entry>
</row>
<row>
<entry>
<para>Deseret</para>
</entry>
<entry>
<para>Devanagari</para>
</entry>
<entry>
<para>Dingbats</para>
</entry>
</row>
<row>
<entry>
<para>EnclosedAlphanumerics</para>
</entry>
<entry>
<para>EnclosedCJKLettersandMonths</para>
</entry>
<entry>
<para>Ethiopic</para>
</entry>
</row>
<row>
<entry>
<para>GeneralPunctuation</para>
</entry>
<entry>
<para>GeometricShapes</para>
</entry>
<entry>
<para>Georgian</para>
</entry>
</row>
<row>
<entry>
<para>Gothic</para>
</entry>
<entry>
<para>Greek</para>
</entry>
<entry>
<para>GreekExtended</para>
</entry>
</row>
<row>
<entry>
<para>Gujarati</para>
</entry>
<entry>
<para>Gurmukhi</para>
</entry>
<entry>
<para>HalfwidthandFullwidthForms</para>
</entry>
</row>
<row>
<entry>
<para>HangulCompatibilityJamo</para>
</entry>
<entry>
<para>HangulJamo</para>
</entry>
<entry>
<para>HangulSyllables</para>
</entry>
</row>
<row>
<entry>
<para>Hebrew</para>
</entry>
<entry>
<para>HighPrivateUseSurrogates</para>
</entry>
<entry>
<para>HighSurrogates</para>
</entry>
</row>
<row>
<entry>
<para>Hiragana</para>
</entry>
<entry>
<para>IdeographicDescriptionCharacters</para>
</entry>
<entry>
<para>IPAExtensions</para>
</entry>
</row>
<row>
<entry>
<para>Kanbun</para>
</entry>
<entry>
<para>KangxiRadicals</para>
</entry>
<entry>
<para>Kannada</para>
</entry>
</row>
<row>
<entry>
<para>Katakana</para>
</entry>
<entry>
<para>Khmer</para>
</entry>
<entry>
<para>Lao</para>
</entry>
</row>
<row>
<entry>
<para>Latin-1Supplement</para>
</entry>
<entry>
<para>LatinExtended-A</para>
</entry>
<entry>
<para>LatinExtendedAdditional</para>
</entry>
</row>
<row>
<entry>
<para>LatinExtended-B</para>
</entry>
<entry>
<para>LetterlikeSymbols</para>
</entry>
<entry>
<para>LowSurrogates</para>
</entry>
</row>
<row>
<entry>
<para>Malayalam</para>
</entry>
<entry>
<para>MathematicalAlphanumericSymbols</para>
</entry>
<entry>
<para>MathematicalOperators</para>
</entry>
</row>
<row>
<entry>
<para>MiscellaneousSymbols</para>
</entry>
<entry>
<para>MiscellaneousTechnical</para>
</entry>
<entry>
<para>Mongolian</para>
</entry>
</row>
<row>
<entry>
<para>MusicalSymbols</para>
</entry>
<entry>
<para>Myanmar</para>
</entry>
<entry>
<para>NumberForms</para>
</entry>
</row>
<row>
<entry>
<para>Ogham</para>
</entry>
<entry>
<para>OldItalic</para>
</entry>
<entry>
<para>OpticalCharacterRecognition</para>
</entry>
</row>
<row>
<entry>
<para>Oriya</para>
</entry>
<entry>
<para>PrivateUse</para>
</entry>
<entry>
<para>PrivateUse</para>
</entry>
</row>
<row>
<entry>
<para>PrivateUse</para>
</entry>
<entry>
<para>Runic</para>
</entry>
<entry>
<para>Sinhala</para>
</entry>
</row>
<row>
<entry>
<para>SmallFormVariants</para>
</entry>
<entry>
<para>SpacingModifierLetters</para>
</entry>
<entry>
<para>Specials</para>
</entry>
</row>
<row>
<entry>
<para>Specials</para>
</entry>
<entry>
<para>SuperscriptsandSubscripts</para>
</entry>
<entry>
<para>Syriac</para>
</entry>
</row>
<row>
<entry>
<para>Tags</para>
</entry>
<entry>
<para>Tamil</para>
</entry>
<entry>
<para>Telugu</para>
</entry>
</row>
<row>
<entry>
<para>Thaana</para>
</entry>
<entry>
<para>Thai</para>
</entry>
<entry>
<para>Tibetan</para>
</entry>
</row>
<row>
<entry>
<para>UnifiedCanadianAboriginalSyllabics</para>
</entry>
<entry>
<para>YiRadicals</para>
</entry>
<entry>
<para>YiSyllables</para>
</entry>
</row>
</tbody>
</tgroup>
</table>

<para>We will see in the next section that W3C XML Schema has introduced an extension to Regular Expressions to specify intersections and that this extension can be used to define the intersection 
between a block and a category in a single pattern facet.
</para>

<note>
<para>Although Unicode blocks seem to be a great idea to restrict text to use a set of characters which you know that you'll be able to print, display, read or store in a database, they have not been designed for this purpose and one must be careful when using them. Here is what John Cowan who is has always fascinated me by his knowledge of Unicode and its more obscure alphabets writes about this topic:</para>
<para><blockquote>It's important to note that Unicode blocks are a very crude mechanism for discrimination: not everything needed to write Greek is in the Greek block, and there are no less than five Latin blocks, one of which (Basic Latin, i.e. ASCII) contains many script-independent symbols. The blocks were originally created solely for internal Unicode organizational purposes, and have spread to the outside world somewhat randomly.</blockquote></para>
<para>The five Latin blocks mentioned by John are BasicLatin, Latin-1Supplement, LatinExtended-A, LatinExtendedAdditional and LatinExtended-B.</para>
</note>

</sect3>

<sect3>
<title>User-defined character classes</title>

<para>These  classes are lists of characters between square
brackets that 
accept <literal>-</literal> signs to
define ranges and a 
leading <literal>^</literal> to negate
the whole list--for instance:
</para>

<programlisting>[azertyuiop]</programlisting>

<para>to define the list of letters on the first row of a French keyboard,</para>

<programlisting>[a-z]</programlisting>

<para>to specify all the characters between
&quot;a&quot; and
&quot;z&quot;,
</para>

<programlisting>[^a-z]</programlisting>

<para>for all the characters that are not between
&quot;a&quot; and
&quot;z,&quot; but also
</para>

<programlisting>[-^\\]</programlisting>

<para>to define the characters &quot;-,&quot;
&quot;^,&quot; and
&quot;\,&quot; or
</para>

<programlisting>[-+]</programlisting>

<para>to specify a decimal sign.</para>

<para>These examples are enough to see that what's between
these square brackets follows a specific syntax and semantic. Like
the regular expression's main syntax, we have a list
of atoms, but instead of matching each atom against a character of
the instance string, we define a logical space. Between the atoms and
the character class is the set of characters matching any of the
atoms found between the brackets.
</para>

<para>We see also two special characters that have a different meaning
depending on their location! The character <literal>-</literal>,
which is a range delimiter when it is between <literal>a</literal>
and <literal>z</literal>, is a normal character when it is just after
the opening bracket or just before the closing bracket
(<literal>[+-]</literal> and <literal>[-+]</literal> are, therefore,
both legal). On the contrary, <literal>^</literal>, which is a
negator when it appears at the beginning of a class, loses this
special meaning to become a normal character later in the class
definition.
</para>
<note>
<para>Even though this is specified as valid by the W3C XML Schema recommendation, it is not supported by all the regular expression engines used by Relax NG processors. The current version of Jing (as I write these lines) do not support <literal>[+-]</literal> nor <literal>[-+]</literal> and it is wiser to escape the character <literal>-</literal> and write either <literal>[+\-]</literal> or <literal>[\-+]</literal>.</para>
<para>Another frequent confusion is about the support of the escape format <literal>#xXX</literal> (such as in <literal>#x2D</literal>. Because this format is used in the W3C XML Schema recommendation to describe characters by their Unicode value, some people have thought that it could be used in regular expressions which is not meant to be the case. If you want to define characters by their Unicode values, you should instead use numeric entities (such as <literal>&amp;#x2D;</literal> if you are using the XML syntax or the syntax for escaping characters in the compact syntax <literal>\x{2D}</literal>. Note that in both cases, this will be replaced by the corresponding character at parse time and that the regular expression engine will see the actual character instead of the escape sequence.</para>
</note>
<para>We   also notice that characters may or
must be escaped: &quot;\\&quot; is used to
match the character &quot;\&quot;. In fact,
in a class definition, all the escape sequences that we have seen as
atoms can be used. Even though some of the special characters lose
their special meaning inside square brackets, they can always be
escaped. So, the following:
</para>

<programlisting>[-^\\]</programlisting>

<para>can also be written as:</para>

<programlisting>[\-\^\\]</programlisting>

<para>or as:</para>

<programlisting>[\^\\-]</programlisting>

<para>since the location of the characters doesn't matter
any longer when they are escaped.
</para>

<para>Within square brackets, the character
&quot;\&quot; also keeps its meaning of a
reference to a Perl or Unicode class. The following:
</para>

<programlisting>[\d\p{Lu}]</programlisting>

<para>is a set of decimal digits (Perl class <literal>\d</literal>) and
uppercase letters (Unicode category
&quot;Lu&quot;).
</para>

<para>Mathematicians have found that three basic
operations are needed to manipulate sets and that these operations
can be chosen from a larger set of operations. In our square
brackets, we already saw two of these operations:
union (the
square bracket is an implicit union of its atoms) and
complement (a leading
<literal>^</literal> realizes the complement of the set defined in
the square bracket). W3C XML Schema extended the syntax of the Perl
regular expressions to introduce a third operation: the difference
between sets. The syntax follows:
</para>

<programlisting>[set1-[set2]]</programlisting>

<para>Its meaning is all the characters in <literal>set1</literal> that do
not belong to <literal>set2</literal>, where <literal>set1</literal>
and <literal>set2</literal> can use all the syntactic tricks that we
have seen up to now.
</para>

<para>This operator can be used to perform
intersections of character classes (the
intersection between two sets A and B is the difference between A and
the complement of B), and we can now define a class for the
<literal>BasicLatin Letters</literal> as:
</para>

<programlisting>[\p{IsBasicLatin}-[^\p{L}]]</programlisting>

<para>Or, using the <literal>\P</literal> construct, which is also a
complement, we can define the class as:
</para>

<programlisting>[\p{IsBasicLatin}-[\P{L}]]</programlisting>

<para>The corresponding definition would be:</para>

<programlisting><![CDATA[
  <define name="BasicLatinLetters">
    <data type="token">
      <param name="pattern">[\p{IsBasicLatin}-[\P{L}]]*</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>
<programlisting><![CDATA[
 BasicLatinLetters = xsd:token {pattern = "[\p{IsBasicLatin}-[\P{L}]]*"}
]]></programlisting>

</sect3>
</sect2>
<sect2>
<title>Oring and Grouping</title>

<para>We have already used a "or" in our 
first 
 example pattern facet when we have written "1|5|15" to say that we wanted to allow either "1", "5" or "15".
</para>

<para>These
&quot;ors&quot; are especially interesting
when used in conjunction with groups.
Groups are complete regular expressions, which are, themselves,
considered atoms and can be used with an optional quantifier to form
more complete (and complex) regular expressions. Groups are enclosed
by brackets (&quot;(&quot; and
&quot;)&quot;). To define a comma-separated
list of &quot;1,&quot;
&quot;5,&quot; or
&quot;15,&quot; ignoring whitespaces between
values and commas, the following pattern facet could be used:
</para>

<programlisting><![CDATA[
  <define name="myListOfBytes">
    <data type="token">
      <param name="pattern">(1|5|15)( *, *(1|5|15))*</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
  myListOfBytes = xsd:token {pattern = "(1|5|15)( *, *(1|5|15))*"}
]]></programlisting>

<para>Note how we have relied on the whitespace processing of the base
datatype (<literal>xsd:token</literal> collapses the whitespaces). We
have not tested leading and trailing whitespaces that are trimmed and
we have only tested single occurrences of spaces with the following
atom:
</para>

<programlisting>run back &quot; * &quot; run back</programlisting>

<para>before and after the comma. 
</para>

</sect2>
</sect1>

<sect1>
<title>Common Patterns</title>

<para>After this overview of the syntax used by
pattern facets, let's see some common pattern facets that you
may have to use (or adapt) in your schemas or just consider as
examples.
</para>

<sect2>
<title>String Datatypes</title>

<para>Regular expressions treat information in its
textual form. This makes them an excellent mechanism for constraining
strings.
</para>

<sect3>
<title>Unicode blocks</title>

<para>Unicode
is a great asset of XML;
however, there are few applications able to process and display all
the characters of the Unicode set correctly and still fewer users
able to read them! If you need to check that your string datatypes
belong to one (or more) Unicode blocks, you can use these pattern facets:
</para>

<programlisting><![CDATA[
  <define name="BasicLatinToken">
    <data type="token">
      <param name="pattern">\p{IsBasicLatin}*</param>
    </data>
  </define>
	
  <define name="Latin-1Token">
    <data type="token">
      <param name="pattern">[\p{IsBasicLatin}\p{IsLatin-1Supplement}]*</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>
<programlisting><![CDATA[
  BasicLatinToken = xsd:token {pattern = "\p{IsBasicLatin}*"}

  Latin-1Token = xsd:token {pattern = "[\p{IsBasicLatin}\p{IsLatin-1Supplement}]*"
]]></programlisting>



<para>Note that such pattern facets do not impose a character encoding on the
document itself and that, for instance, the
<literal>Latin-1Token</literal> datatype would validate instance
documents using UTF-8, UTF-16, ISO-8869-1 or other encoding. (This
assumes the characters used in this string belong to the two Unicode
blocks <literal>BasicLatin</literal> and
<literal>Latin-1Supplement</literal>.) In other words, working on the
lexical space, i.e., after the transformations have been done by the
parser, these pattern facets do not control the physical format of the
instance documents.
</para>

</sect3>

<sect3>
<title>Counting words</title>

<para>The pattern facet can be used to limit the number of
words in a text. To do so, we will define an atom,
which is a sequence of one or more
&quot;word&quot; characters
(<literal>\w+</literal>) followed by one or more nonword characters
(<literal>\W+</literal>), and control the number of occurrences of this atom. If
we are not very strict on the punctuation, we also need to allow an
arbitrary number of nonword characters at the beginning of our value
and to deal with the possibility of a value ending with a word
(without further separation). One of the ways to avoid any ambiguity
at the end of the string is to dissociate the last occurrence of a
word to make the trailing separator optional:
</para>

<programlisting><![CDATA[
  <define name="story100-200words">
    <data type="token">
      <param name="pattern">\W*(\w+\W+){99,199}\w+\W*</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
  story100-200words= xsd:token {pattern = "\W*(\w+\W+){99,199}\w+\W*"}
]]></programlisting>


</sect3>

<sect3>
<title>URIs</title>

<para>The <literal>xsd:anyURI</literal> datatype
doesn't care about
&quot;absolutizing&quot; relative URIs and in some cases it
is wise to impose the usage of absolute URIs, which are easier to
process. Furthermore, it can also be interesting for some
applications to limit the accepted URI schemes. This can easily be
done by a set of pattern facets such as:
</para>
<programlisting><![CDATA[
  <define name="httpURI">
    <data type="anyURI">
      <param name="pattern">http://.*</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
  httpURI= xsd:anyURI {pattern = "http://.*"}
]]></programlisting>



</sect3>
</sect2>
<sect2>
<title>Numeric and Float Types</title>

<para>While  numeric types aren't
strictly text, pattern facets can still be used appropriately to constrain
their lexical form.
</para>

<sect3>
<title>Leading zeros</title>

<para>Getting rid of  leading zeros is quite simple but
requires some precautions if we want to keep the optional sign and
the number &quot;0&quot; itself. This can be
done using pattern facets such as:
</para>
<programlisting><![CDATA[
  <define name="noLeadingZeros">
    <data type="integer">
      <param name="pattern">[+-]?([^0][0-9]*|0)</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
  noLeadingZeros= xsd:integer {pattern = "[+-]?([^0][0-9]*|0)"}
]]></programlisting>



<para>Note that in this pattern facet, we chose to redefine all the lexical rules
that apply to an integer. This pattern facet would give the same lexical
space applied to a <literal>xsd:token</literal>
datatype as on a <literal>xsd:integer</literal>. We could also have
relied on the knowledge of the base datatype and written:
</para>

<programlisting><![CDATA[
  <define name="noLeadingZeros">
    <data type="integer">
      <param name="pattern">[+-]?([^0].*|0)</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
  noLeadingZeros= xsd:integer {pattern = "[+-]?([^0].*|0)"}
]]></programlisting>


<para>Relying on the base datatype in this manner can produce simpler
pattern facets, but can also be more difficult to interpret since we would
have to combine the lexical rules of the base datatype to the rules
expressed by the pattern facet to understand the result.
</para>

</sect3>

<sect3>
<title>Fixed format</title>

<para>The maximum number of digits can be fixed using

<literal>xsd:totalDigits</literal> and <literal>xsd:fractionDigits</literal>. However, these facets are only maximum
numbers and work on the value space. If we want to fix the format of
the lexical space to be, for instance,
&quot;DDDD.DD&quot;, we can write a pattern facet
such as:
</para>
<programlisting><![CDATA[
  <define name="fixedDigits">
    <data type="decimal">
      <param name="pattern">[+-]?\.{4}\..{2}</param>
    </data>
  </define>
]]></programlisting>

<para>Or:</para>

<programlisting><![CDATA[
  fixedDigits= xsd:decimal {pattern = "[+-]?\.{4}\..{2}"}
]]></programlisting>

</sect3>
</sect2>
<sect2>
<title>Datetimes</title>

<para>Dates  and time have complex lexical representations.
Patterns can give developers extra control over how they are used.
</para>

<sect3>
<title>Time zones</title>

<para>The  time zone support of W3C XML Schema is
quite controversial and needs some additional constraints to avoid
comparison problems. These pattern facets can be kept relatively simple
since the syntax of the datetime is already checked by the schema
validator and only simple additional checks need to be added.
Applications which require that their datetimes specify a time zone
may use the following template, which checks that the time part ends
with a &quot;Z&quot; or contains a sign:
</para>

<programlisting><![CDATA[
  <define name="dateTimeWithTimezone">
    <data type="dateTime">
      <param name="pattern">.+T[^Z+-]+</param>
    </data>
  </define>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
 dateTimeWithTimezone= xsd:dateTime {pattern = ".+T[^Z+-]+"}
]]></programlisting>


<para>Simpler, applications that want to make sure that none of their
datetimes specify a time zone may just check that the time part
doesn't contain the characters
&quot;+&quot;,
&quot;-&quot;, or
&quot;Z&quot;:
</para>

<programlisting><![CDATA[
  <define name="dateTimeWithoutTimezone">
    <data type="dateTime">
      <param name="pattern">.+T[^Z+-]+</param>
    </data>
  </define>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
 dateTimeWithoutTimezone= xsd:dateTime {pattern = ".+T[^Z+-]+"}
]]></programlisting>



<para>In these two datatypes, we used the separator
&quot;T&quot;. This is convenient, since no
occurrences of the signs can occur after this delimiter except in the
time zone definition. This delimiter would be missing if we wanted to
constrain dates instead of datetimes, but, in this case, we can
detect the time zones on their &quot;:&quot;
instead:
</para>

<programlisting><![CDATA[
  <define name="dateWithTimezone">
    <data type="date">
      <param name="pattern">.+[:Z].*</param>
    </data>
  </define>
  <define name="dateWithoutTimezone">
    <data type="date">
      <param name="pattern">[^:Z]</param>
    </data>
  </define>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
dateWithTimezone= xsd:date {pattern = ".+[:Z].*"} 
dateWithoutTimezone= xsd:date {pattern = "[^:Z]"} 
]]></programlisting>


<para>Applications may also simply impose a set of time zones to use:</para>

<programlisting><![CDATA[
  <define name="dateTimeInMyTimezones">
    <data type="dateTime">
      <param name="pattern">.+(\+02:00|\+01:00|\+00:00|Z|-04:00)</param>
    </data>
  </define>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
 dateTimeInMyTimezones= xsd:dateTime {
 	pattern = ".+(\+02:00|\+01:00|\+00:00|Z|-04:00)"
 }
]]></programlisting>



<para>We can also constrain <literal>xsd:duration</literal> to a couple of subsets that have a complete sort order. The
first datatype will consist of durations expressed only in months and
years, and the second will consist of durations expressed only in
days, hours, minutes, and seconds. The criteria used for the test can
be the presence of a &quot;D&quot; (for day)
or a &quot;T&quot; (the time delimiter). If
neither of those characters are detected, then the datatype uses only
year and month parts. The test for the other type cannot be based on
the absence of &quot;Y&quot; and
&quot;M&quot;, since there is also an
&quot;M&quot; in the time part. We can test
that, after an optional sign, the first field is either the day part
or the &quot;T&quot; delimiter:
</para>

<programlisting><![CDATA[
  <define name="YMduration">
    <data type="duration">
      <param name="pattern">[^TD]+</param>
    </data>
  </define>
  <define name="DHMSduration">
    <data type="duration">
      <param name="pattern">-?P((\d+D)|T).*</param>
    </data>
  </define>
]]></programlisting>

<para>or:</para>

<programlisting><![CDATA[
YMduration= xsd:duration {pattern = "[^TD]+"} 
DHMSduration= xsd:duration {pattern = "-?P((\d+D)|T).*"} 
]]></programlisting>



</sect3>
</sect2>
</sect1>

</chapter>
